An Algebraic Characterization of Security of 
Cryptographic Protocols 



Manas K. Patra and Yan Zhang 

Department of Computing and Mathematics, University of Western Sydney, 
Locked Bag 1797, Penrith South DC, NSW 1797 
Australia 



Abstract. Several of the basic cryptographic constructs have associated 
algebraic structures. Formal models proposed by Dolev and Yao to study 
the (unconditional) security of public key protocols form a group. The 
security of some types of protocols can be neatly formulated in this al- 
gebraic setting. We investigate classes of two-party protocols. We then 
consider extension of the formal algebraic framework to private-key pro- 
tocols. We also discuss concrete realization of the formal models. In this 
case, we propose a definition in terms of pseudo-free groups. 
Keywords/Topics: security, public key cryptosystem, free and pseudo-free 
groups and monoids. 



1 Introduction and Background 

The present paper explores some algebraic structures inherent in several classes 
of security protocols. Such structures have been known to exist. For example, 
the set of possible messages over some alphabet A constitute a free monoid 
A*. The encryption and decryption operations must be inverse of each other. If 
we consider them as mappings A* — > A* they form a group. Moreover, many 
encryption schemes are based on some well-known algebraic structures. The RSA 
encryption is a bijective map Z n — > Z n , where Z n is the ring of integers modulo 
n. So we have, on the one hand, formal models of classes of protocols which carry 
algebraic structures and on the other concrete realizations of these models based 
on some sets with inherent algebraic structures. One of the basic issues addressed 
in this paper is the notion of security of protocols in the algebraic setting. In the 
formal model where we assume perfect encryption the security is unconditional. 
Hence, it can be breached due to a faulty design of protocols. In the concrete 
model however the encryption is based on the assumption that certain tasks 
are computationally infeasible. In this case, the security can be compromised 
due to a faulty design or some hidden relations among the basic operators. 
Although protocols based on PKC are believed to be secure against passive 
attacks, an improperly designed protocols may be compromised by an active 
adversary, as first pointed out by Needham and Schroeder |NS78j . The analysis 
of all possible such attacks requires some level of abstraction and formalization. 
Such a formalization was first given in the seminal work of Dolev and Yao DY82 



(referred to as DY) . The class of protocols discussed in DY are two-party cascade 
protocols in which two users exchange messages back and forth. Other notable 
early works dealing with the formal approach to security include [BAN89ILow 96 . 

An alternative approach to security is the computational approach. Infor- 
mally, a protocol is considered secure if it is computationally infeasible for the 
adversary to acquire any useful information |Gol01IAR02] . The computational 
approach is more difficult in proving security of protocols. Starting with the work 
of |AR02| there has been extensive work to relate the two views of cryptography. 
In |AR02j the authors first give the formal framework for some cryptographic 
primitives. In terms of security, their main result roughly translates to the fol- 
lowing: if we can show that a protocol, formally an expression, is equivalent 
to another expression over a fixed string then the protocol is secure since it is 
infeasible for the adversary to distinguish between the actual plaintext and an 
arbitrary bit string. Thus a formal system is sound if formal indistinguishabil- 
ity (FI) implies computational indistinguishability (CI). The converse (CI => 
FT) is called the completeness of the formal system. It has been proved for the 
Abadi-Rogaway formal system under some extra assumptions MW04a]. The 
works AR02 MW04aJ dealt with symmetric (private) key encryptions and pas- 
sive adversaries and in MW04bJ the authors prove soundness of a formal system 
similar to |AR02j for public key cryptosystem with active adversaries. The work 
|MW04b deals with issues that are closest to the current work. 

In this work we take a fresh look at the DY model. We investigate algebraic 
structures associated with a class of protocols based on public key cryptosys- 
tems. We observe that the model defined by strings of operators can be given 
the structure of group called the Dolev-Yao (DY) group. The main results of 
this paper are characterization of the security of the protocols in these group 
structures. Specifically, we show that a set of elements (strings) defining the pro- 
tocol is insecure if and only if they contain a subgroup. This is strictly true in 
the abstract setting when we assume that there are no special relations among 
the elements- the DY group is free. In a concrete realization there will be some 
relations among the group elements. We propose extensions of the notion of se- 
curity in terms of pseudo-free groups rather than free groups. We also consider 
extension to private key cryptosystems. 

We first review the Dolev-Yao model. One defines the abstract setting of a 
protocol in terms of some basic operations (encryption, decryption, nonces etc.). 
These operations form a monoid. Then a protocol is simply a sequence of words, 
the elements of the monoid. The security of a protocol is defined in terms of 
these words. Specifically, we show that a set of elements (strings) defining the 
protocol is insecure if and only if they contain a subgroup. We consider first the 
simple cascade protocols where the message texts are encrypted and decrypted 
straight without further operations like nonces. In this case the monoid turns 
out to be a group and a protocol is insecure if and only the elements defining it 
form a subgroup. Next, we consider protocols with nonces (name-stamps, date- 
stamps etc.). The algebraic characterization is trickier here because some of the 
operations are undefined in a real implementation. We show that even in this case 



we can sensibly define a monoid of operations and characterize protocol security 
in terms of some algebraic condition. We use the algebraic characterizations to 
prove some general theorems on secure and insecure protocols. We apply these 
results to some well-known protocols. We also discuss the concrete realization 
of the cryptosystems. We analyze the implication on security in this situation. 
The problem of security in an arbitrary realization is undecidable since it can 
be reduced to the word problem [Rot 9 5] . The final section discusses possible 
extensions of the definitions and methods. 

2 The Dolev-Yao model 

In this section we review the essentials of the model proposed by Dolev and 
Yao. The first assumption is that we do not concern ourselves with the details 
of the public key cryptographic system. Further, we assume that we have a fi- 
nite set of symbols £ = {Ei, Ei, ■ ■ . , E n } where n is an integer. Informally, n 
denotes the number of users in the network and Ei represents the public encryp- 
tion function of the i th user. Similarly we have another set T> = {D\, . . . , D n } 
representing the private decryption function of the users. For example, if Ki 
and K[ are the public and private keys of user i then Ei(M) — E(M,Ki) and 
Di(M) = D(M, K'A, where E and D are the respective encryption and decryp- 
tion functions and M is the message text. We also add another operator, / the 
"identity" operator. In general the encryption and decryption schemes need not 
be same for all users but they must satisfy E1D1 = D; L E; L = I. We simply treat 
them as letters from some alphabet. For each pair of users define the sets 
Aij — {Ei, Ej, Di} Informally, Aij represents the set of operators available to 
user i in a two-party exchange between itself and user j. A two party cascade 
protocol is finite sequence of strings {oti, c*2, ■ ■ ■ , cx r } and {j3x,^2, ■ ■ ■ ,Pr'} where 
on G A*j and Pi S A*i, 1 < i, j • < n and r 1 = r — 1 or r. Intuitively, users i and 
j can use any number of layers of encryption and decryption and thus the set of 
operations available are included in £ U V. The definition of cascade protocols is 
a consequence of the following assumption on the protocols [DY82J . 

1. It is a perfect public key system. Hence: 1. the functions Ei are strictly one 
way: they are unbreakable, 2. the public directory is secure: the Ei are fixed 
once for all, 3. everyone has access to all the encryption functions Ei, 4. only 
user i knows Di. 

2. In the two-party protocol only the two parties concerned are involved in the 
communication; the assistance of a third party is not needed. 

3. The protocols are uniform, that is, the same format is used by any pair of 
legitimate users. 

4. Next we model the behavior of the adversary. We assume that the adversary 
is capable of active attacks. Specifically: 1. the adversary can intercept any 
message passing through the communication channels;2. he is a legitimate 
user and thus can initiate a dialog with other users; 3. he can successfully 
impersonate another user when necessary. 



We assume that the above assumptions are valid for any protocol (not just 
cascade protocols) unless stated otherwise. 

Next we describe the formal model for the protocols. Let x, y be variables 
ranging through the set J„ = {0,1,..., n). A two-party cascade protocol is given 
by a pair of sequences 

{a 1 (x,y),a 2 (x,y),a r (x,y)} and {0x(x, y), (3 2 (x, y), (3 r > (x, y)} (1) 
di(x,y) e A xy and (3i{x,y) <E A yx (2) 

Further, define the sequences 

Ni(x,y) =a 1 (x,y) N 2 (x,y) = 1 (x,y)a 1 (x,y) 
N 2 k-i(x,y) = a k (x,y)N 2k ^ 2 (x,y) N 2k (x,y) = (3 k N 2k ^ 1 (x, y) 

The intuition behind this abstract definition is the following. User x initiates 
the dialog with y by applying a\{x, y) to the message M G {0,1}*. Then, y 
responds with the application of 0i(x,y), x follows with a 2 (x,y) and so on. In 
round k (k > 1) user x sends the message N 2k _iM and in turn, receives the 
message N 2k M. For example, in the simple protocol discussed later we have 
ai(l, 2) = E 2 , and /3i(l, 2) = E\D 2 Let V be a two-party cascade protocol. Let 
s be any user name (the adversary) and 

A(s) = £ U {D s }, r 2 = {a 2 {x, y)\ for all x ^ y and i > 2} and 
r-3 = {Pi(x, y)\ for all x ^ y and i > 1} 

Next we define the security of a protocol. 

Definition 1 A protocol V is insecure if there is some string X £ ii(s) UT 2 UI3 
such that XN k (i,j) — e for some k (e denotes the empty string). 

Sec DY82 for the motivation for this definition is as follows. If the protocol is 
insecure then the secret message can eventually be obtained by the adversary. 



3 The Dolev-Yao group for cascade protocols 

We start this section with some standard algebraic definitions [Rot95]. A semi- 
group is set S with a binary operation or product o that is associative (a o 
(60 c) = (a o &)). A monoid is a semigroup {S, 0} with an identity element e 
(eoa = doe = a). A group is a monoid M such that every a 6 M has an inverse 
a -1 (a o a -1 = a -1 o a = e). Below we suppress the symbol o for the product. 
We have seen above that for cascade protocols the available operators are from 
£ U T>. The set £* (the Kleene closure of £ ) is the set of words, including the 
empty word, formed by the alphabet £ . Now consider the free group generated 
by the set £ [MKS76] . We recall the free group construction. Let A be a set (the 
alphabet). Let A -1 be another set, disjoint from A such that there is a bijective 
correspondence a <-> a between the two. We write A^ 1 = {a _1 |a e A}. Let e 



be the empty string. Then we define a product on the set Sa = {A U A^ 1 )* by 
concatenation (a-fi = cr/i) along with the relations act -1 = a _1 a = e. That is, we 
replace aa^ 1 and a~ 1 a by e in any string. More formally, define an equivalence 
relation ~ between two strings a and fi as: a ~ y« if can be obtained from 
ex by insertion or deletion of strings of the form aa^ 1 , a~ 1 a and e. Then the set 
F(A) = Sa/ ~j the set of equivalence classes is a group. For details see MKS76 . 
For convenience, we continue to write the members of F(A) as elements of Sa 
rather than the equivalence class. For a free monoid we have only the set A and 
the relation e. The essential property of a free group or monoid F(A) over the set 
A is that any mapping of the set A into a group G can be uniquely extended to 
a group homomorphism (see jMKS76j for details). Recall that a homomorphism 
between two monoids is a mapping that preserves the identity and products. 
A homomorphism between two groups is a homomorphism of the underlying 
monoids that preserves inverses. A submonoid A of a monoid M is a subset with 
identity that is closed under products. We call F{£ ) the DY group. Further, we 
use Di and E~ l interchangeably. A concrete realization of the DY group is given 
by the action of encryption and decryption operators on {0, 1}*, the set of bi- 
nary strings. Thus, if Kj,, and Pj are i's public and private key respectively then 
Ei(m) = E(m,Ki) and -Dj(m) = D(m,Pi). We note that a concrete realization 
of a free group may result in more relations. For example, for a commutative 
group we have the relations ab ~ ba. We further mention that a particular re- 
alization realization of the DY group in the RSA encryption scheme is distinct 
from the RSA group |Riv04| . In general, the latter is commutative while the 
former is not. 

Let us consider an example discussed in |DY82] . User i sends j a message 
m (i, Ej(m),j) and then j sends back the message (J, Ei(m), i). This protocol is 
very easily broken. The adversary, henceforth denoted by s, intercepts the first 
message from i and sends it to j. Then j sends the message (j, E s (m), s). The 
adversary decrypts the message using D s . It is easy to verify that in this case 
the the monoid generated sets i~i = {D s } and J2 = {E s DjEj} a subgroup of 
DY. We will see that this is a general phenomenon for insecure protocols. 

3.1 An algebraic characterization of security 

In this section we come to the main theme of this work. Dolev and Yao gave 
a characterization of the secure cascade protocols in terms of properties of the 
strings a.i(x,y) and f3j(x,y). We prove an equivalent characterization in the 
algebraic setting of the DY group. We can then deduce their characterization. 
In the following, the word generate will always imply the multiplicative set (a 
monoid) . 

Theorem 1 Let V be a two-party cascade protocol. Assume that the parties 
involved have names 1 and 2 and the adversary is s. Then, with the notation as 
above, V is insecure if and only if there is a set T C {E\, E<i, E s , D s } C A(s) 
such that one of the following condition holds. 

1. The set {ai(x,y)} U T generates a subgroup of DY multiplicatively. 



2. The set T\jr 2 (x, y) LlZ^x, y);x,y € {1, 2, s} generates a nontrivial subgroup 
ofDY. 

where rj(x,y) denotes the set r 2 with specific users x and y. 

Proof. Let us first note that the first condition takes care of a rather trivial 
situation. It can only come about if the user x initiates the conversation by 
sending the message without an encryption or if she applies her own decryption 
operator! In any case, it is clear that the protocol is insecure. Next, suppose the 
second condition holds. Then the set T U r 2 (x,y) U r%(x,y) a subgroup S. In 
particular, Ey , E 2 S S. Hence, there is a string A e S such that XN = e since 
the latter is the identity element of the group. It follows from the definition [T] 
that the protocol is insecure. This proves the sufficiency of the condition. 

To prove necessity of the condition assume that the protocol is insecure. Then 
there is some string A such that XNi = e, i > 1 First, suppose that i = 1 and 
Ni = ct\ does not contain E\ or E 2 . Then we must have ct\ — e or for some 
integer k. In the first case, we obtain the trivial subgroup by choosing T to be 
empty set and in the second case we choose T = {Ei}. In either case, the first 
condition of the theorem is satisfied. 

Now let Ni, i > 1 satisfy the above equation. Suppose i = 2j is even (the 
proof for the odd case is similar). Then 

N 2j (l, 2) = ((3,(1, 2)^(1, 2) • ■ ■ a 2 (l, 2)/3 1 (l, 2))ai(l, 2) 
= 0j-(l,2)ai(l,2) and 
A = ar 1 (l,2)^T 1 (l,2) 

By assumption, A 6 (A(s) UT 2 (x, y) UT 3 (x, y))* = H. Let H' = H U {a x (l, 2)}. 
Clearly we may restrict to the set {l,2,s} of users. Observe first that any 
N(x,y) is of the form E^ 1 E^ 1 E^ 2 E° y 2 ■ ■ ■ E x i m Ej m where i r and j r are inte- 
gers. Recall that we identify E~ x = D x . Suppose that all the exponents of 
E±, and E 2 in the expansion of 7V2j(l,2) are non-negative. We may assume 
that at least one of them, say that of Ei, is positive (otherwise there is noth- 
ing to prove). Then by successive application of Ei or E 2 we conclude that 
E^ 1 is in H . From the definition of the sets E 2 and Fj, we can interchange the 
role of Ei and E 2 and we conclude that E^ 1 is also a member of H. Choose 
T = {E s ,Ei,E 2 ,D s }. Then, Tur 2 l,2Ur 3 l,2 generates a subgroup. Hence, we 
may assume that N 2 j(l, 2) contains negative powers of Ei, i = 1,2. In any case 
wehave JV 2i (l,2) = ^(1, 2)ai(l, 2) and A = otf^l, 2)^>J 1 (1, 2). As ^-(1,2) £ H 
we conclude that a^ 1 £ H. Let a^ 1 = E~ n Ep 1 E^ 12 Ep 2 ■ ■■Ep m E~ Jm E H 
Where ik,jk are integers. We recall that cti may contain only Ei,E 2 , or Di. 
Thus, no jk can be negative. We have assumed that not all of them are zero 
for otherwise we are back to the first condition. Therefore, we may write a^ 1 — 
E^ D^ 1 E\ 2 D 2 j 2 ■ ■ ■ E\ m Dp We assume that none of the exponents in the middle 
(that is, ji,i 2 , ■ ■ ■ ,i m ) are zero and consider several cases. As ap belongs to 



the set H, it must be of the form 



0^(1,2) = a'"! *(l,2) l 3^-- i '' ) (l,2)^ 1 ^ 1 

Q! (4,-,«2)(i ; 2)/3 {b ^-- b i\l, 2)El 2 E% 2 ■ ■ ■ 

where cx^--' a ^(l,2) = a^(l,2) • 0^(1,2) and ^-'^ (1,2) = 

Px l (l, 2) • • • Pi 1 (1, 2) etc.. Now, the set H contains cti(x,y), i > 2 and /3j(x,y) 
for all x ^ y and all Ei. Hence we may replace ai(l,2) with a^s,2), (3j(l,2) 
with Pj(s,2) and £i with E s . This substitution will replace all E\ and £>i by 
E s and Z? s respectively. Now we may apply E s , D s and E2 in appropriate order 
to obtain D2 in H . We next consider a7 (2, 1) and arguing as above we conclude 
that Di G and that the semigroup generated by 7J is a subgroup. 

We note that in case of insecure protocols the subgroup generated by H is 
the full group generated by the encryption operators {E\, E%, E s } of the three 
parties concerned : the initiator, the intended receiver and the adversary. The 
theorem gives an abstract algebraic characterization of security. For practical 
purposes we would want a syntactic characterization in terms of the strings of 
operators. For this we start with a definition. 

Definition 2 Let S = E^E^? ■ ■ ■ EV° k be a string with i\, . . . ,%k integers and 
jii ■ ■ ■ ijk £ {lj ■ • ■ 1 n } in reduced form. For an integer r in the set {1, . . . , n} 
define the r-index of S to be the sequence of integers (r(l),r(2), . . . ,r(m)) which 
appear as nonzero exponents of E r in S. We say that the r-index of S is negative 
if the largest integer in the sequence is negative. 

If the r index of a string S is negative then all the exponents of E r (there 
must be at least one) are negative. That is, only D r appears in 5*. Such strings 
are unbalanced as per [DY82] . Let us also say that r — index is zero if no powers 
of E r appears in the string. Now we can state the second characterization of 
insecure protocols. 

Theorem 2 Let V a two-party cascade protocol. Assume that the legitimate 
parties have names 1 and 2 and the former initiates the conversation. Then V 
is insecure if and only if one of the following holds: 

1. The 2-index of a\(\, 2) is zero and the 1-index of a\(l, 2) is zero or negative. 

2. There exists some a,;, i > 2 whose 1-index is negative. 

3. There exists some (3i, i > 1 whose 2-index is negative. 

Proof. Sufficiency. If the first condition above is satisfied then it is easy to see 
that the first condition in Theorem [T] holds. Suppose now that the second or the 
third condition holds. We can use arguments similar to those in the previous 
theorem to show that E\ and E2 are in S the semigroup generated by H — 

U r 2 (x, y) U r 3 (x, y), x, y G {1, 2, s}. 
Necessity. The proof of necessity is rather long. We only outline the steps. Sup- 
pose V is insecure. From Theorem[T]we infer that either the first condition holds 



or S is a subgroup. If the first condition holds then clearly the 2-index of ai(l, 2) 
is zero and the 1-index of a\(l, 2) must be zero or negative. We may thus assume 
that S is a subgroup. Then El 1 £ S. Write £f 1 is a product of aj(x, y), i > 2, 
Pi(x,y), i > 1 and the £Vs. We use induction on the length 2 of such product. 
The case I = 1 is clear. Let I = k. That is, E^ 1 = 7$ where j £ H and is in S. 
By assumption, none of the factors in <P have negative r-index for r G {1, 2, s}. 
Now OLki}, j) (resp. (3k(i,j)) cannot have negative j (resp. i) index. Next show 
that if 71,72 £ H have nonnegative r-index (r = 1,2) then their product 7172 
also has nonnegative r-index. This is straightforward but lengthy. By assump- 
tion each of the generators of S have nonnegative r— index r £ {1,2}. Hence, 
as 7 and <5 have positive r-index for r = 1, 2 and so does 7^, a contradiction. 

The theorem yields the following corollary in some concrete realization of the 
cryptosystem. We recall that there may extra relations among the generators in 
any such realization. Let these relations be given by the set R C F(£ ) where 
we put any x £ R equal to e. Two strings in F{£) are equivalent if they can be 
reduced to each other by insertion or deletion of elements from R. Then we have 

Corollary 1 A concrete realization of a two-party protocol is insecure if and 
only if each string in the equivalence classes of an, i > 1 and 0j , j > 1 has 
nonnegative 1 and 2 index. 

3.2 Algebraic characterization of security of general protocols 

In this section we will consider protocols with nonces (e.g. name-stamp). In 
the cascade protocols the structure of the plain text message itself played no 
role in the protocol. A name-stamp protocol uses the structure of the mes- 
sage to improve security. We use the notation as above. Now each user has 
more operations available. We have first the operation of nonce A x for user x: 
A X (M) — Mx. We also have the partial inverse 6 X , the deletion operator, that 
is, S X A X (M) = M . The problem is that it only makes sense to apply 6 X imme- 
diately after A x (after reduction in EiS and -Djs). In fact, in [DY82] and other 
treatments DEK82 EG83J the application of 6 X is undefined in all other cases. 
However, for the algebraic structures we require that all products be well-defined. 
Let O x = {Ey,D x ,A y ,6y\y any user } be the set of operators available to user 
x. Let A be the set of operators A x and A, the set of S x s. We postulate the 
following relations:. E X D X = D X E X = e and S X A X — e. Note that in this case we 
no longer have group since A X 8 X =/= e. 

Definition 3 A two-party name-stamp protocol is given by the following se- 
quences of strings: ai(x,y) £ ({E x , E y , D x }LlAuA)*, %{x,y) £ ({E x ,E y ,D y }U 
AUA)* 

We will assume that the protocol is well-defined, that is, there are no illegal oper- 
ations of 8 X . Let O — U x O x be the set of operators of all users. Let Go be the free 



monoid generated by O. We are identifying E x 1 with D x . We define AT (x, y) = 
e,Ni(x,y) = ai(x ) y),N 2 (x,y) = Pi(x,y)a>i(x,y), . . .,N 2 j-i(x,y) = aj(x,y)N 2 j- 2 (x,y) 
and N2j{x,y) = (3j(x,y)N2j-i{x,y) as before. We define a protocol to be inse- 
cure if there is a string 7 G (-F{ U U fj)* such that jNi(x,y) = e for some 
i > 1 where 

^(s) = {^ B ,-Ba,I»«,^x,«x| s G {a,b,s}} 
r 2 = {ai(x, y)\x, y G {a, 6, s} and i > 2} 
= {A (a, 2/) 1^,2/ G {a, 6, s} and t > 1} 

The motivation for the above definition of insecurity is similar to the case of 
cascade protocols. Excluding the trivial case (when the initiator a sends the first 
string without encryption!) we state the algebraic characterization of security of 
these general protocols. 

Theorem 3 A name- stamp protocol is insecure if and only if (T[ UT^U ^3)* 
contains the subgroup of Gq freely generated by {E a , Ei,, E s }. 

Proof. (Sketch) We observe first that, as in Theorem [1] the condition for inse- 
curity is equivalent to requiring that the string a±(a, b) has an inverse. Clearly, 
the condition is sufficient since we can generate D a = E~ x and — E^ 1 and 
hence the inverse of any string. 

The necessity of the condition can proved using arguments similar to Theo- 
remQ] We write cei(a, 6) _1 as a product of elements from r[ Uf^U 7^. Then by 
appropriate changes a — > s or b — > s we can obtain D a and Db- 

The theorem implies, in particular, that the empty string e is in (T{ UT^ U/^) + , 
where for any set of strings S, S + = S* — {e} . The security of two-party ping-pong 
protocol is therefore equivalent to a decision problem for a regular language: is 
the empty string a member of the language. For our case the problem is tractable. 
It is fairly straightforward to write an algorithm for the decision problem for the 
language (r[ U r' 2 U whose time complexity is bounded by polynomial in 
the length of the protocol. An efficient algorithm is given in [DEK82 . 
Let us consider some special protocols. 

Proposition 1 Let a protocol V be given by the following strings. a%,0i = 
7ia|j~ 1 ,Q;2 = A*27]" 1 '/^2 = l2l JL 2 1 ' ' ' suc h that ci!i,7j and fa have nonnegative 
1 and 2 index, are not empty, do not contain any 5 X and have their left-most 
symbol appropriate name-stamp A x . Here a~ 1 denotes the left inverse of a. Then 
V is secure. 

Proof. (Sketch) Suppose V is insecure. Then there exist v\, v 2> ■ ■ ■ , Ufc G (rlUT^U 
7^) such that D\ = V\ ■ ■ ■ Vk- Then one of the v^s must be some cti — fa r y~\. 
But the right-most symbol of 7^ is a S x . Hence, it must cancel. In fact, all 
the inverses must cancel. We are left with strings 74 's and fij's. But these have 
nonnegative 1-index and from the previous section one cannot obtain D\ with 
these generators. 



We can similarly show that if in some protocol V we have some aj(l,2), i > 
2), j > 1) such that the substrings on the left and right of the left-most 
62(61) have negative l-index(2-index) then the protocol is insecure. We only have 
to consider aj(l,s) and cancel appropriate symbols using D S ,A S , and E\. 

3.3 Examples, Extensions, and Concrete Realizations 

Consider now a simplified variant of Needham-Schroeder authentication protocol 
|Low96j . We have a x {l,2) = E 2 A 1 ,f3 ( l,2) = E X 6 2 D 2 and a 2 (l,2) = E 2 D 1 . In 
detail, user 1 stamps its nonce and sends the string to 2 using the latter's public 
key encryption E 2 . User 2 then decrypts the message and sends it back to 1 
using its public encryption and 1 decrypts the message and sends it to 2 after 
encryption. We see at once that the protocol is insecure because 0:2(1, 2) — E 2 D\ 
has negative 1-index. We observe that the reason it is insecure is because there 
is no nonce in stage 2. Hence, if we modify the protocol |Low96l with ai(l, 2) = 
E2A1, /9/1, 2) = E\A 2 6\D 2 and a' 2 (l, 2) = E 2 6 2 Di from the above proposition it 
follows that the protocol is secure. On the other hand, following protocol [DY82 
is insecure: ai(l,2) = E 2 A 1 E 2 , /3i(l, 2) = .Ei^AjchAj, since in /3i(l,2) the 
substrings to left and right of <5i have negative 2-index. We therefore observe that 
with the use of above propositions we can eliminate large classes of protocols 
as insecure. Although we do not have a necessary and sufficient criterion for 
security (as in the case of ping-pong protocols) we can write efficient algorithms 
to verify security. These are essentially rewriting algorithms in groups |Sim94j . 

We investigated the algebraic structures arising out of protocols based on 
public or asymmetric key cryptosystem. Can we extend this to private or sym- 
metric key cryptography. In case of, two party protocols the answer is yes. If 
users 1 and 2 share a private key then we set E\ = E 2 and remove E\ from 
adversary's set of operations r (see the previous section). The security of the 
protocols is defined as above. 

A (concrete) realization of an abstract protocol is a map <f> : Go ~ * G which 
is monoid homomorphism. Here Go is the free monoid on the set O of op- 
erations available to all users and G is some monoid. Any map from O to 
G can be uniquely extended to a homomorphism 4> : Go — * G. In general, 
G will satisfy some extra relations. For example, if G is finite then for any 
x £ G, x' G ' = e Then, the security criteria of Theorem [3] is inadequate since 
any subset of G will generate nontrivial subgroups. An example is the cyclic 
subgroup {E\,Ef, . . . , i?| G ' = e}. Hence, we must modify the security condition. 
Our proposal is to require the relevant groups be only pseudo-free |Riv04IMic05j 
instead of free. Informally, a group G is pseudo-free if any polynomial time prob- 
abilistic algorithm designed to find relations in G that are not satisfied in a free 
group will succeed with only negligible probability. Let V be a two-party proto- 
col and let r be the set of operators (in reduced form) available to an adversary 
as in the preceding sections. Then <j>(r) may contain non-trivial groups. Suppose 
all these groups are pseudo-free. Then any special relations that the adversary 
may try to exploit can only be found with negligible probability by any feasi- 
ble algorithm. We note that the security of a protocol may be compromised in 



two ways. First, the adversary may break the cryptosystem itself, for example, 
by finding an efficient algorithm to factorize integers in RSA-based cryptosys- 
tem. The second way is to exploit some weakness in the protocol itself as in the 
Needham-Schroeder protocol. Both cases are covered by the following definition. 

Definition 4 A protocol V is insecure if and only if one of the following holds. 

1. In the free group r , the set of operations available to the adversary generate 
a nontrivial subgroup. 

2. The maximal subgroup contained in the monoid generated by r[ U U .£3 in 
a family of concrete realizations of the encryption and decryption operators 
is not pseudo-free. 

If the basic public key cryptosystem is RSA then in general the encryption 
operators Ei are based on different moduli and the messages may have to be split 
into blocks of appropriate size before each encryption. The operators Ei are quite 
complicated and form a non-abelian group. In the ElGamal encryption scheme 
[E1G85] the encryption operator is a map E a : Z* —* Z* where E a (m) = mg Xak 
. All operations are modulo p, g is a primitive generator of Z*, g andg^" are 
publicly known. The number k is randomly chosen by b and g k is publicly known. 
The adversary does not know k or x a and hence E a . This is similar to the case of 
private key cryptosystem since we have to remove E a from the set of operations 
available to adversary. If all users use the same p the group is abelian. However, 
if they choose different primes the messages have to be block and the resulting 
realization of the DY group is non-abelian in general. 

4 Discussion 

In this work we presented an algebraic characterization of security of public key 
protocols. We may question the advantages of the algebraic characterization. 
First, there are theoretical advantages. We have at our disposal powerful tech- 
niques of group theory. To prove some fact in the setting of free groups we can 
define a homomorphism from the free group to another (not necessarily free) 
group which has a simpler structure. For example, in Theorem [5] we defined the 
notion of r-index and stated that it is positive for the product of two strings 
whose r-index is positive. The proof is given by induction and a tedious case by 
case consideration on the structure of the two strings. It is possible to give a 
group theoretic proof by defining a homomorphism to another group via some 
defining relations. Secondly, there are practical advantages too. Sometimes, of- 
ten computations and rewriting in groups is simpler and we have at our disposal 
several computational tools |Sim94| . 

This work is an attempt to give a new, algebraic perspective on security and 
there is still a lot of ground to be covered. Can we extend the formal algebraic 
characterization to other protocols? An essential requirement for group structure 
is that all the operations be invertible. For example, could also include operations 



like pairing. We then just have the structure of a monoid, as in the case of name- 
stamp protocols and we have seen that these can be dealt with in an algebraic 
setting. We aim to deal with these issues in the future. 
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